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ABSTRACT 

This document is designed to be used with the "Users' 
Guide". It is wrii^ten for practitioners who are interested in meeting 
the federal regulations governing the evaluation of Title VII 
projects but do not have evaluation training. Six types of bilingual 
projects are required to fulfill the evaluation requirements 
specified in the June 19, 1986 Bilingual Education Regulations 
(Sections 500e50, 500.51, 500.52): (1) transitional (basic) projects; 
(2) developmental ^oasic) projects; (3) special alternative 
instructional (basic) projects; (4) academic excellence projects; (5) 
special population projects; and (6) family English literacy 
projects. Part 1 of this document is a copy of the June 19, 1987 
evaluation regulations. Each requirement ii presented again in part 
2, followed by recommendations of what Titlv^ VII grantees should do 
to satisfy the regulation. Suggested evaluation activities are 
briefly described. (SM) 
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INTRODUCTION 



This document is an abridged version of Volume I of the Bilingual Education 
Evaluation System Users' Guide. It is written for practitioners who do not have 
training in evaluation but who are interested in meeting the Federal regulations 
governing the evaluation of their Title VII projects. Six types of bilingual projects 
are required to fulfill the evaluation requirements specified in the June 19, 1986 
Bilingual Education Regulations (Sections 500.50, 500.51, 500.52): transitional 
(basic) projects, developmental (basic) projects, special alternative instructional 
(basic) projects, academic excellence projects, special population projects, and 
family English literacy projects. 

Part 1 of this Abbreviated Recommendations is a copy of the June 19, 1987 
evaluation regulations. In Part 2, each requirement is presented again, followed by 
our recommendations on what Title VII grantees should do to satisfy it. Suggested 
evaluation activities are described only very briefly. For this reason, persons who 
will actually conduct the evaluation should read Volumes I and II of the Users' 
Guide, and if necessary, seek assistance from their regional Evaluation Assistance 
Center. 
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PART 1 



Subpart F: What Evaluation Requirements Must Be Met by a Recipient? 
§500.50 What evaluation requirements apply to a grantee? 

(a) This section establishes a comprehensive design of the general 
evaluation requirements and standards that a grantee funded 
unaer programs authorized under part A of Title VII must 
meet in carrying out an annual evaluation of a project which 
provides instructional services to limited English proficient 
persons. 

(b) A grantee's evaluation must comply with the following 
requirements: 

(1) A grantee's evaluation design must include a measure 
of the educational progress of project participants when 
measured against an appropriate nonproject com- 
parison group. 

(2) A grantee's evaluation design must meet the following 
technical standards: 

(i) Representativeness of evaluation fitidings. The 
evaluation results must be computed so thit the 
conclusions apply to the persons, schools or 
agencies served by the projects. 

(ii) Reliability and validity of evaluation instruments 
and procedures. The evaluation instruments used 
must consistently anj accurately measure 
progress toward accomplishing the objectives of 
the project, and must be appropriate considering 
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factors such as the age, grade, language, degree 
of language fluency and background of the per- 
sons served by the project. 

(iii) Evaluation procedures that minimize error. The 
evaluation procedures must minimize error by 
providing for proper administration of the 
evaluation instruments, at twelve-month testing 
intervals, accurate scoring and transcription of 
results, and the use of analysis and reporting 
procedures that are appropriate for the data ob- 
tained from the evaluation. 



(iv) Valid measurement of academic acnievement. 
The evaluation procedures must provide objec- 
tive measures of the acanemic achievement of 
participants related tO English language 
proficiency, native or second language 
proficiency (for programs of developmental 
bilingual education), and other subject matter 
areas. 



(3) 



(i) A grantee's evaluation must provide information 
on the academic achievement of ~ 



(A) Current participants in the project, who 
are - 



(1) Children who are limited English 
proficient; and 
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(2; Children whose language is 
English; and 



(B) Children who were formerly served in the 
project as limited English proficient, have 
exited from the program, and are now in 
English language classrooms. 

(ii) This information must include- 

(A) The amount of time (in years or school 
months, as appropriate) the participants 
received instructional services in the 
project, and as appropriate, in another in- 
structional setting. 

(B) The participants' progress in achieving 
English language proficiency and, for 
programs of developmental bilingual 
education, progress in another language; 
and 

(C) The former participants' academic 
progress in English language classrooms. 

[Authority: 20 U.S.C. 3243] 

(Approved by the Office of Management and Budget under control number 1885- 
0003) 

§500.5 1 What evaluation information must a grantee collect? 

In carrying out the aii^iual evaluation under §500.50, a grantee shall 
collect information on- 
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(a) 



The educational background, needs, and competencies of the 
limited English proficient persons served by the project; 



(b) The specific educational activities undertaken pursuant to the 
project; 

(c) The pedagogical materials, methods, and techniques utilized in 
the program: 

(d) With respect to classroom activities, the relative amount of in- 
structional time spent with students on specific tasks; and 

(e) The educational and professional qualifications, including lan- 
guage competencies, of the staff responsible for planning and 
operating the project. 

[Authority: 20U.S.C3243] 

(Approved by t^e Office of Management and Budget under control number 1885- 
0003) 



§500.52 What information must a grantee report to the Secretary? 

A grantee shall report to the Secretary annually, the information col- 
lected in §500.51 and an evaluation of the overall progress of the 
project including the extent of educational progress achieved through 
the project measured, as appropriate by- 

(a) Tests of academic achievement in English language arts, and 
for programs of developmental bilingual education, second 
language arts; 

(b) Tests of academic achievement in subject matter areas; and 
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(c) Changes in the rate of student- 

( 1) Grade-retention; 

(2) Dropout; 

(3) Absenteeism; 

(4) Referral to or placement in special education 
classes; 

(5) Placemen! in programs for the gifted and 
talented; and 

(6) Enrollment in postsecondary education 
institutions. 

[Authority: 20 U.S.C 3242] 

(Approved by the Office of Management and Budget under control number 1885- 
0003) 
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PART 2 



Requirement: [§500.50(b)(l)] 

A grantee's evaluation design must include a measure of the educational progress of 
project participants when measured against an appropriate nonproject comparison 
group. 

Recommpndations: 

■ Use the gap-redrction design to measure the educational progress of 
project students relative to one of the following two nonproject groups: 

the nonproject grade mates of the project students. 

the 50th percentile of the national norms (requires the use of a stand- 
ardized achievement test). 

The gap-reduction design is described in Volume I, Chapter VII of the 
Bilingual Education Evaluation System Users' Guide. Computer software 
is available to perform the calculations. Manual computational proce- 
dures are presented in Volume II of the Users' Guide, Appendix H. 

- Pretest late entering students as near to the time of their entry as pos- 
sible. 

- Posttest students you know will be moving away as close to their 
departure date as possible, 

- If fewer than 90% of the students served have both pre- and posttest 
scores^ describe how those students differed from those students with 
complete data. 
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Requirement: [§500.50(b)(2)(i)] 

The evaluation results must be computed so that the conclusions apply to the persons, 
schools or age*' served by the projects. 

Recommendations: 

■ Consider that all students who participatea in the project for 100 days or 
more were served by the project. 

■ Make an effort to collect data from all project participants (and non- 
project comparison group students, if any). Schedule some make-up tests 
for participants who were absent when the pre- and or posttests were ad- 
ministered. 
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Requirement: [§500.50(b)(2)(ii)J 

The evaluation instruments used must consistently and accurately measure progress 
toward accomplishing the objectives of the project, and must be appropriate considering 
factors such as the age, grade, language, degree of language fluency, and background of 
the persons served by the project. 

Recommendations: 

■ Select achievement tests that match your curriculum. 

Use standardized achievement tests whenever possible. 

Examine candidate tests for a match with your curriculum on an item- 
by-item basis. Categorize each item as (a) having been previously 
covered, (b) to be covered in the coming year, (c) to be covered in fol- 
lowing years, or (d) not in the curriculum. 

Select any test in which at last 30-40% of the items match your cur- 
riculum (category b) and some items fall into categories (a) and (c). 

Use subtests alone if this improves the match with \ Mr curriculum. 

■ Select tests with appropriate difficulty levels. If you have not used the test 
before, use your own judgment or administer a 'locator" test to determine 
its difficulty. After you have administered the test, you can use the results 
to determine its difficulty level. 

A test is too difficult if any student does not know the answer to a 
single question. 

A test is too easy if any student knows the answers to all of the items. 

If the test is too easy or too difficult, you 
should correct the problem with functional 
level testing, i.e., testing students at a level dif- 

9 
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ferent from the one the publisher has assigned 
to their grade. 

You can administer a lower level at pretest 
time and the recommended level at posttest, if 
this solves the difficulty problem. 

If your test is too difficult, you can also correct 
the problem by modifying the test. 

Keep changes to a minimum. 

Appropriate changes include clarifying test in- 
structions, extending time limits, and replacing 
words or phrases that are unfamiliar to stu- 
dents and that are not relevant to the content 
area being tested. 

If you modify a standardized test, use the same 
level for both pre- and posttesting, and don't 
use the test results for assessing students' 
status in relation to the norm group. 

If you modify a test, keep the modifications the 
same for pretest and posttest. 

■ If you do not have the flexibility to select an appropriate achievement test, 
you should still review the test you will use both for curriculum match and 
for difficulty level. Describe your findings in your report. 

■ Coach students in test-taking skills. Do as much of this as possible before 
the pretest. 

■ Avoid developing a new test without the help of a testing expert. Conduct 
an item analysis of any test you develop. 

■ Use multiple measures whenever possible for documenting project im- 
pact. Check to see if scores from: 



project entry and/or exit tests, 

state or district testing programs, and/or 
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teacher-made tests 

tend to confirm or conflict with the results obtained from your primary 
evaluation instrument. 
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Requirement: [§500. 50 (b)(2) (Hi)] 

The evaluation procedures must minimize error by providing for proper administration 
of the evaluation instruments^ at twelve-month testing intervals, accurate scoring and 
transcription of results, and the use of analysis and reporting procedures that are ap- 
propriate for the data obtained from the evaluation. 

Recommendation s: 

■ Every year, train test administrators to: 

Follow written instructions exactly at both pretest and posttest times. 

Note student behaviors during testing which indicate invalid test 
scores (e.g., illness, copying, random or pattern responding). 

■ Avoid stakeholder bias. If possible, use data collectors, processors, and 
analysts who do not have an interest in the success of the program. If 
stakeholders are used to conduct the evaluation activities, remind them 
that, with annual data collection, one year's posttest is the next year's 
pretest. 

■ Prepare students for testing. 

Tell them where, when, and why they will be tested. 

Encourage parents to make sure students are well rested the night 
before anc have a good breakfast on the day of testing. 

Give practice tests and/or teach test taking skills. 

■ Prepare the testing setting. 

The space should be clean, quiet, and large enough to provide ample 
space between students. 
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Desks or tables should be large enough to hold booklets and answer 
sheets. 

Use the same testing space for both regular and make-up testing. 

■ When using standardized achievement tests, test within two weeks of the 
test's norming date. 

■ Make sure scoring and score conversions are accurate. 

Use a machine scoring service if possible, particularly if you need 
score conversions. Clean answer sheets before having them scored. 

If scoring is performed by hand or answers are transferred, have a 
second person check for accuracy on a random 5% of answer sheets. 
If an error is found, check all sheets. 

Check a random 5% of scores again after data are entered into the 
computer or onto data summary forms. 

■ Discard invalid data. Data may be judged to be invalid if: 

"Problems" were observed during testing. 

Students did not follow the testing instructions. 

Scores are extremely high or low in comparison to the average group 
performance. 

■ In analyzing test results, follow prescribed procedures exactly. 

Use medians rather than means if there is evidence that tests were too 
difficult or too easy. 
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■ In reporting results, be sure to describe any unusual events or cir- 
cumstances that may have affected test results. 
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Requirement: [§500.50(b)(2)(iv)J 

The evaluation procedures must provide objective measures of the academic achieve- 
ment of participants re' '^^d to English language proficiency, native or second language 
proficiency (for programs of developmental bilingual education), and other subject mat- 
ter areas. 

Recommendations: 

■ Use appropriate tests to measure achievement in different subject areas* 

Measure proficiency in English with tests written in English. 

Measure proficiency in LI with tests written in LI. 

Test non-language academic subjects in the language of instruction, 
unless students are dominant in English. 

■ When testing in LI, you should attempt to locate a published test. If none 
can be found, translating a test of known quality will usually be preferable 
to developing a new test. 

■ Use objective measures of academic achievement such as standardized 
achievement tests. Do not use subjective measures such as a teachers' 
ratings of student performance. 

■ Make sure the assessment instruments, data collectors, testing settings, 
and data collection procedures are the same at pretest and posttest times. 

■ Standardize administration procedures for tests that come without 
prescribed procedures. 
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Requirement: [§500.50(b)(3)J 



(i) A grantee's evaluation must provide infonnation on the academic achievement 



(B) Children who were formerly served in the project as limited English profi- 
cient, have exited from the program, and are now in English language 
classrooms 

(ii) This information must include- 

(A) The amount of time (in years or school months, as apptopriate) the par- 
ticipants received instructional services in the project, and as appropriate, 
in another instructional setting; 

(B) The participants' progress in achieving English language proficiency and, 
for programs of developmental bilingual education, progress in another 
language; and 

(C) The former participants' academic progress in English language class- 



Recommendations: 

■ For both current and former project participants, report: 

the number of years or school months they received instructional serv- 
ices from the project. 

The number of years or school months they received instructional 
services in another instructional setting (e.g., mainstream class, 
migrant program. Chapter 1). 

■ For current project participants, report: 



of: 



(A) 



Current participants in the project, who are: 



1. 

Z 




rooms. 
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progress (preferably gap reduction) in achieving proficiency in English 
(usually reading readiness or reading, although oral language 
proficiency, writing, or other language arts skill may sometimes be 
more appropriate). 

- for programs of developmental bilingual education, progress in LI 
language skills. 

For /ormer project participants, report progress in English language 
classrooms-preferably using gap-reduction measures. 

Rank order students on the basis of their classroom grades at the 
beginning ano end of the year to see if former project students are 
falling behind, keeping pace with, or catching up to their mainstream 
classmates. 

If scores are available, compare former project students* performance 
on district administered tests to that of their mainstream classmates. 
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Requirement: [§500,51(a)J 

A grantee shall collect information on the educational background, needs, and com- 
petencies of the limited English proficient persons served by the project; 



Recommendations: 

■ Collect information on project students' 
- age. 

grade level, 
first language, 
ethnicity (language group), 
language used in the home. 

proficiency in LI (including literacy, if appropriate), 
proficiency in English. 

socioeconomic status (e.g., participation in National School Lunch 
Program. 
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Requirement: [§500.51(b)] 

A grantee shall collect information on the specific educational activities undertaken 
pursuant to the project; 

Recommendations? 

■ Collect information on: 

subject areas included in project instruction. 

major curriculum objectives. 
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Requirement: [§500.51(c)J 

A grantee shall collect information on the pedagogical materials, methods, and tech- 
niques utilized in the program; 

Recommendations: 

■ Collect information on: 

key instructional features. 

project materials different from those used in mainstream classrooms, 
percentage of instructional time devoted to English language arts 
percentage of instructional time devoted to LI language arts, 
instructional areas taught in LI. 

length of time students are expected to remain in the project. 
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Requirement: f§500.51(d)] 



A grantee shall collect information on...the relative amount of instructional time spent 
with students on specific tasks; 

Recommendations: 

■ For each major curriculum objective, collect information on the hours per 
year at each grade level devoted to the objective. 
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Requirement: [§500.51(e)J 

A grantee shall collect information on the educational and professional qualifications, 
including language competencies, of the staff responsible for planning and operating the 
project. 

Recommendations: 

■ Collect information on the: 

academic preparation of each staff member, including level of educa- 
tion and credentials and certificates. 

teaching experience of each staff member, including years of ex- 
perience in monolingual and in bilingual classrooms. 

language abilities of each staff member, including languages under- 
stoo'^' and spoken, degree of fluency, and whether English is spoken 
with a heavy accent. 
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Requirement: [§500.52] 

A gruittee shall report to the secretary annually, the infonnation collected in §500.51 
and an evaluation of the overall progress of the project including the extent of educa- 
tional progress achieved through the project measured, as appropriate, by- 

(a) Tests of academic achievement in English language arts, and for 
programs of developmental bilingual education, second language arts; 

(b) Tests of academic achievement in subject matter areas; and 

(c) Changes in the rate of student- 

(1) Grade-retention; 

(2) Dropout; 

(3) Absenteeism; 

(4) Referral to or placement in special education classes; 

( 5) Placement in programs for the gifted and talented; and 

(6) Enrollment in postsecondary education institutions. 

Recommendations: 

■ To test academic achieveineii* in English language arts, standardized read- 
ing readiness or reading tests should be preferred. Oral language 
proficiency tests or other language arts tests may be used if they provice a 
better match to your curriculum. 

■ Measure academic achievement using more than one type of test. If pos- 
sible, report student progress as measured by standardized achievement 
tests, teacher made tests, and curriculum unit tests. 

1 Nontest data, like test data, must be collected on a before-and-after basis 
(or early during the students' project participation versus later). Data on 
low frequency events (e.g., placement in programs for the gifted and 
talented) may require pooling data across years in order to make reliable 
assessments of change. 
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